When mobile communication terminals typified by mobile phones are developed, a mobile communication terminal test device is used to test whether the mobile communication terminal under development operates as designed and whether the mobile communication terminal normally operates according to a communication standard.
A test scenario in which the operation sequence and communication sequence of the mobile communication terminal test device are defined is created in advance and stored, and the mobile communication terminal test device operates as a pseudo mobile network, simulating the behaviour of multiple elements in the network such as base station, mobile switching centre, packet gateway servers according to the test scenario. Then, the tester (developer) uses the mobile communication terminal test device to communicate with the mobile communication terminal to be tested and checks the operating state of the mobile communication terminal and a communication state between the mobile communication terminal and the mobile network with reference to log information during communication, thereby testing whether the mobile communication terminal conforms to required standards and can communicate and behave normally.
With the evolution of mobile communication technology, an increasing number of communication standards exist. The staged rollout of new communication technology demands that mobile communication terminals are able to communicate with a wide range of mobile communication technologies, protocols and base station standards. As such, mobile communication terminal test devices are designed to simulate a plurality of communication bands, protocols and standards so that a mobile communication terminal can be tested against modern standards and also regression tested against older standards. Examples of protocols a mobile communication terminal may be tested against includes Long Term Evolution (LTE), LTE-Advanced (a next-generation standard of LTE), third generation partnership project (3GPP or simply 3G), Edge, and GSM (global system for mobile communications) to name a few.
The requirement to develop a test scenario which can evaluate whether a mobile communication terminal is correctly operating across a wide range of standards and communication protocols necessarily requires a complex set of test procedures which may be produced by different test developer groups and then combined into a single test scenario.
Typically, these test scenarios have multiple execution paths to accommodate the variety of communication standards and protocols, expected responses the mobile communication terminal and test conditions. A test scenario, when executed according to a particular test configuration, forms a test operation. Typical mobile communication terminal test devices perform test operations and indicate a pass/fail for each such operation.
The intent of the test designer is that executing a test scenario with a range of configuration information would enable complete testing through all execution paths of a test scenario. In practice, it is impossible to determine whether a test scenario will execute through all possible execution paths until testing is performed on a mobile communication terminal. This presents test designers with a challenge in verifying a mobile communication terminal has been fully tested via execution of numerous test operations. The indication of pass/fail for each test operation is not sufficient for a test designer to determine whether complete testing has occurred.